[
https://issues.apache.org/jira/browse/FLINK-27399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufan Sheng updated FLINK-27399:
--------------------------------
Description:
The Pulsar connector didn't use the consuming position from the checkpoint.
They just commit the position to Pulsar after the checkpoint is complete. And
the connector starts to consume messages from Pulsar directly by the offset
stored on the Pulsar subscription.
This causes the test could be failed in some situations. The start cursor
(position on Pulsar) would be reset to the wrong position, which caused the
results didn't match the desired records.
h2. How to fix this issue
Change the start position seeking mechanism from Pulsar consumer API to Pulsar
admin API. Don't reset the start position when the topic has a subscription.
h2. This issue fixes
# FLINK-23944
# FLINK-24872
# FLINK-25815
# FLINK-25884
# FLINK-26177
# FLINK-26237
# FLINK-26721
# FLINK-27833
was:
The Pulsar connector didn't use the consuming position from the checkpoint.
They just commit the position to Pulsar after the checkpoint is complete. And
the connector starts to consume message from Pulsar directly by the offset
stored on the Pulsar subscription.
This causes the test could be failed in some situations. The start cursor
(position on Pulsar) would be reset to the wrong position, which caused the
results didn't match the desired records.
This issue fixes
# FLINK-23944
# FLINK-24872
# FLINK-25815
# FLINK-25884
# FLINK-26177
# FLINK-26237
# FLINK-26721
# FLINK-27833
> Pulsar connector didn't set start consuming position correctly
> --------------------------------------------------------------
>
> Key: FLINK-27399
> URL: https://issues.apache.org/jira/browse/FLINK-27399
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
> Affects Versions: 1.14.4, 1.15.0, 1.16.0
> Reporter: Yufan Sheng
> Assignee: Yufan Sheng
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.16.0, 1.15.2, 1.14.6
>
>
> The Pulsar connector didn't use the consuming position from the checkpoint.
> They just commit the position to Pulsar after the checkpoint is complete. And
> the connector starts to consume messages from Pulsar directly by the offset
> stored on the Pulsar subscription.
> This causes the test could be failed in some situations. The start cursor
> (position on Pulsar) would be reset to the wrong position, which caused the
> results didn't match the desired records.
> h2. How to fix this issue
> Change the start position seeking mechanism from Pulsar consumer API to
> Pulsar admin API. Don't reset the start position when the topic has a
> subscription.
> h2. This issue fixes
> # FLINK-23944
> # FLINK-24872
> # FLINK-25815
> # FLINK-25884
> # FLINK-26177
> # FLINK-26237
> # FLINK-26721
> # FLINK-27833
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
