[
https://issues.apache.org/jira/browse/FLINK-30413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-30413:
-----------------------------------
Fix Version/s: pulsar-4.0.0
(was: 1.17.0)
> Drop Share and Key_Shared subscription support in Pulsar connector
> ------------------------------------------------------------------
>
> Key: FLINK-30413
> URL: https://issues.apache.org/jira/browse/FLINK-30413
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Pulsar
> Affects Versions: 1.17.0
> Reporter: Yufan Sheng
> Priority: Critical
> Fix For: pulsar-4.0.0
>
>
> A lot of Pulsar connector test unstable issues are related to {{Shared}} and
> {{Key_Shared}} subscription. Because this two subscription is designed to
> consume the records in an unordered way. And we can support multiple
> consumers in same topic partition. But this feature lead to some drawbacks in
> connector.
> 1. Performance
> Flink is a true stream processor with high correctness support. But support
> multiple consumer will require higher correctness which depends on Pulsar
> transaction. But the internal implementation of Pulsar transaction on source
> is record the message one by one and stores all the pending ack status in
> client side. Which is slow and memory inefficient.
> This means that we can only use {{Shared}} and {{Key_Shared}} on Flink with
> low throughput. This against our intention to support these two subscription.
> Because adding multiple consumer to same partition can increase the consuming
> speed.
> 2. Unstable
> Pulsar transaction acknowledge the messages one by one in an internal
> Pulsar's topic
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
