[
https://issues.apache.org/jira/browse/FLINK-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14733814#comment-14733814
]
Hilmi Yildirim edited comment on FLINK-2624 at 9/7/15 3:05 PM:
---------------------------------------------------------------
Unfortunately, there is another problem. RabbitMQ delivers messages to the
consumers and deletes messages if they are acknowledged. RabbitMQ does deliver
a single message to only one consumer and waits for the acknowledgement until
the connection is lost, e.g. the consumer crashes. In this case RabbitMQ sends
the messages which are not acknowledged to other consumsers. For example,
consumer c received messages m1,m2 and m3. If a consumer has acknowledged only
m1 without m2 and m3 and he crashes, then m2 and m3 will be delivered to other
conumsers.
This means in our case: If a worker node crashes during the snapshotting
without acknowledging all processed messages, then the state of this worker
node is useless. Furthermore, the messages the consumer has acknowledged before
he crashed, are deleted.
was (Author: hilmiyildirim):
Unfortunately, there is another problem. RabbitMQ delivers messages to the
consumers and deletes messages if they are acknowledged. RabbitMQ does deliver
a message to only one consumer and waits for the acknowledgement until the
connection is lost, e.g. the consumer crashes. In this case RabbitMQ sends the
messages which are not acknowledged to other consumsers. For example, consumer
c received messages m1,m2 and m3. If a consumer has acknowledged only m1
without m2 and m3 and he crashes, then m2 and m3 will be delivered to other
conumsers.
This means in our case: If a worker node crashes during the snapshotting
without acknowledging all processed messages, then the state of this worker
node is useless. Furthermore, the messages the consumer has acknowledged before
he crashed, are deleted.
> RabbitMQ source / sink should participate in checkpointing
> ----------------------------------------------------------
>
> Key: FLINK-2624
> URL: https://issues.apache.org/jira/browse/FLINK-2624
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
> Affects Versions: 0.10
> Reporter: Stephan Ewen
> Assignee: Hilmi Yildirim
>
> The RabbitMQ connector does not offer any fault tolerance guarantees right
> now, because it does not participate in the checkpointing.
> We should integrate it in a similar was as the {{FlinkKafkaConsumer}} is
> integrated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
