[
https://issues.apache.org/jira/browse/FLINK-32550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jiangchunyang updated FLINK-32550:
----------------------------------
Description:
when i use flink datastream:pulsar-connector. I found that something error when
consume data from pulsar.
Send data to pulsar every second through the program. When there is a
partitioned-topic with 3 partitions in pulsar, use the Exclusive consumption
mode for consumption (verified, it is not a data problem, because there is no
abnormal data in normal tasks)
start A 3-parallel task consumes this topic, and everything is normal; but when
I use a single-parallel task to consume this topic, an exception occurs when
some data is deserialized, and the avro format data is used, so an error is
reported (Length is negative: -52); When I create a partitioned-topic with a
single partition, I use a single parallelism task to consume, and there is no
such problem, all the data is normal, and there is no task error.
I checked the logs and found that the consumer and topic-partition are
allocated as expected.
Summarize:
3 topic-partitions and 1 consume work with exception (3 partitions are
allocated to a single consumer)
3 topic-partitions and 3 consumers work normally
1 topic-partition 1 consumer works normally
importance:
As long as the number of consumers is less than the number of pulsar
topic-partitions, only one consumer will be processing data, and other
consumers will not be able to process data properly
was:
when i use flink datastream:pulsar-connector. I found that something error when
consume data from pulsar.
Send data to pulsar every second through the program. When there is a
partitioned-topic with 3 partitions in pulsar, use the Exclusive consumption
mode for consumption (verified, it is not a data problem, because there is no
abnormal data in normal tasks)
start A 3-parallel task consumes this topic, and everything is normal; but when
I use a single-parallel task to consume this topic, an exception occurs when
some data is deserialized, and the avro format data is used, so an error is
reported (Length is negative: -52); When I create a partitioned-topic with a
single partition, I use a single parallelism task to consume, and there is no
such problem, all the data is normal, and there is no task error.
I checked the logs and found that the consumer and topic-partition are
allocated as expected.
Summarize:
3 topic-partitions and 1 consume work with exception (3 partitions are
allocated to a single consumer)
3 topic-partitions and 3 consumers work normally
1 topic-partition 1 consumer works normally
> Single-parallel task consumes multi-partition data exception in
> pulsarSource-connector
> --------------------------------------------------------------------------------------
>
> Key: FLINK-32550
> URL: https://issues.apache.org/jira/browse/FLINK-32550
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
> Affects Versions: 1.16.0
> Environment: flink 1.16.0
> pulsar 2.11.1
> Reporter: jiangchunyang
> Priority: Major
>
> when i use flink datastream:pulsar-connector. I found that something error
> when consume data from pulsar.
> Send data to pulsar every second through the program. When there is a
> partitioned-topic with 3 partitions in pulsar, use the Exclusive consumption
> mode for consumption (verified, it is not a data problem, because there is no
> abnormal data in normal tasks)
> start A 3-parallel task consumes this topic, and everything is normal; but
> when I use a single-parallel task to consume this topic, an exception occurs
> when some data is deserialized, and the avro format data is used, so an error
> is reported (Length is negative: -52); When I create a partitioned-topic with
> a single partition, I use a single parallelism task to consume, and there is
> no such problem, all the data is normal, and there is no task error.
> I checked the logs and found that the consumer and topic-partition are
> allocated as expected.
> Summarize:
> 3 topic-partitions and 1 consume work with exception (3 partitions are
> allocated to a single consumer)
> 3 topic-partitions and 3 consumers work normally
> 1 topic-partition 1 consumer works normally
> importance:
> As long as the number of consumers is less than the number of pulsar
> topic-partitions, only one consumer will be processing data, and other
> consumers will not be able to process data properly
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
