[
https://issues.apache.org/jira/browse/CAMEL-16914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Otavio Rodolfo Piske updated CAMEL-16914:
-----------------------------------------
Description:
[^FakeDataProducer.java][^MessageLossReproducer.java]I noticed that when using
the KafkaIdempotentRepository, sometimes it seems that it is losing or
corrupting the ID information, causing loss of messages.
When reading the topic used for publishing the idempotency data with kafkacat I
get:
{code:java}
ERROR: Failed to format message in fake-data-inserts [1] at offset 39: expected
key end-of-input, but 2 bytes remaining : terminating{code}
As a result, this causes the loss of messages because the consumers cannot go
further than the corrupted record.
I found this while using Camel K and Strimzi (on a setup with 3 Kafkas using 1
topic with 3 partitions).
I couldn't test w/ 3.12 yet, but I suspect it would also have the same problem.
was:
I noticed that when using the KafkaIdempotentRepository, sometimes it seems
that it is losing or corrupting the ID information, causing loss of messages.
When reading the topic used for publishing the idempotency data with kafkacat I
get:
{code:java}
ERROR: Failed to format message in fake-data-inserts [1] at offset 39: expected
key end-of-input, but 2 bytes remaining : terminating{code}
As a result, this causes the loss of messages because the consumers cannot go
further than the corrupted record.
I found this while using Camel K and Strimzi (on a setup with 3 Kafkas using 1
topic with 3 partitions).
I couldn't test w/ 3.12 yet, but I suspect it would also have the same problem.
> camel-kafka: possible corruption of idempotency messages when using
> KafkaIdempotentRepository
> ----------------------------------------------------------------------------------------------
>
> Key: CAMEL-16914
> URL: https://issues.apache.org/jira/browse/CAMEL-16914
> Project: Camel
> Issue Type: Task
> Components: camel-kafka
> Affects Versions: 3.11.1
> Reporter: Otavio Rodolfo Piske
> Priority: Major
> Attachments: FakeDataProducer.java, MessageLossReproducer.java
>
>
> [^FakeDataProducer.java][^MessageLossReproducer.java]I noticed that when
> using the KafkaIdempotentRepository, sometimes it seems that it is losing or
> corrupting the ID information, causing loss of messages.
>
> When reading the topic used for publishing the idempotency data with kafkacat
> I get:
>
> {code:java}
> ERROR: Failed to format message in fake-data-inserts [1] at offset 39:
> expected key end-of-input, but 2 bytes remaining : terminating{code}
>
> As a result, this causes the loss of messages because the consumers cannot go
> further than the corrupted record.
>
> I found this while using Camel K and Strimzi (on a setup with 3 Kafkas using
> 1 topic with 3 partitions).
>
> I couldn't test w/ 3.12 yet, but I suspect it would also have the same
> problem.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
