[
https://issues.apache.org/jira/browse/FLINK-30413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17677395#comment-17677395
]
Matthias Pohl commented on FLINK-30413:
---------------------------------------
{quote}Can you please provide backports for 1.16 and 1.15? We've seen test
instabilities (the once collected under FLINK-28352) in older versions as well.
Those are not fixed with your PR. Example:
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44691&view=logs&j=a5ef94ef-68c2-57fd-3794-dc108ed1c495&t=2c68b137-b01d-55c9-e603-3ff3f320364b&l=27466]
which is caused by FLINK-27917
{quote}
I just found out why this test failure popped up:
{{PulsarOrderedPartitionSplitReaderTest}} failed but I mixed it up with
{{PulsarUnorderedPartitionSplitReaderTest}}.
> Drop Shared and Key_Shared subscription support in Pulsar connector
> -------------------------------------------------------------------
>
> Key: FLINK-30413
> URL: https://issues.apache.org/jira/browse/FLINK-30413
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Pulsar
> Affects Versions: 1.16.0, 1.17.0, 1.15.3
> Reporter: Yufan Sheng
> Assignee: Yufan Sheng
> Priority: Critical
> Labels: pull-request-available
> Fix For: pulsar-4.0.0
>
>
> A lot of Pulsar connector test unstable issues are related to {{Shared}} and
> {{Key_Shared}} subscription. Because this two subscription is designed to
> consume the records in an unordered way. And we can support multiple
> consumers in same topic partition. But this feature lead to some drawbacks in
> connector.
> 1. Performance
> Flink is a true stream processor with high correctness support. But support
> multiple consumer will require higher correctness which depends on Pulsar
> transaction. But the internal implementation of Pulsar transaction on source
> is record the message one by one and stores all the pending ack status in
> client side. Which is slow and memory inefficient.
> This means that we can only use {{Shared}} and {{Key_Shared}} on Flink with
> low throughput. This against our intention to support these two subscription.
> Because adding multiple consumer to same partition can increase the consuming
> speed.
> 2. Unstable
> Pulsar transaction acknowledge the messages one by one in an internal
> Pulsar's topic. But it's not stable enough to get it works. A lot of pending
> issues in Flink JIRA are related to Pulsar transaction and we don't have any
> workaround.
> 3. Complex
> Support {{Shared}} and {{Key_Shared}} subscription make the connector's code
> more complex than we expect. We have to make every part of code into ordered
> and unordered way. Which is hard to understand for the maintainer.
> 4. Necessary
> The current implementation on {{Shared}} and {{Key_Shared}} is completely
> unusable to use in Production environment. For the user, this function is not
> necessary. Because there is no bottleneck in consuming data from Pulsar, the
> bottleneck is in processing the data, which we can achieve by increasing the
> parallelism of the processing operator.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
