[
https://issues.apache.org/jira/browse/FLINK-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14736955#comment-14736955
]
Hilmi Yildirim commented on FLINK-2624:
---------------------------------------
I have implemented checkpointing for RabbitMQ. But the checkpointing does only
work if there is only one consumer for a queue. I think that the only way to
enable multi consumer checkpointing is to save all received messages. Messages
can be deleted if they are included in a completed snapshot. The main issue is
that, unlike Kafka, RabbitMQ is not designed for "Big Data" consumers.
See:
RMQSource:
https://github.com/HilmiYildirim/flink/blob/RabbitMQNonParallelSnapshotting/flink-staging/flink-streaming/flink-streaming-connectors/flink-connector-rabbitmq/src/main/java/org/apache/flink/streaming/connectors/rabbitmq/RMQSource.java
NonParallelConnectorSource:
https://github.com/HilmiYildirim/flink/blob/RabbitMQNonParallelSnapshotting/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/functions/source/NonParallelConnectorSource.java
> RabbitMQ source / sink should participate in checkpointing
> ----------------------------------------------------------
>
> Key: FLINK-2624
> URL: https://issues.apache.org/jira/browse/FLINK-2624
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
> Affects Versions: 0.10
> Reporter: Stephan Ewen
> Assignee: Hilmi Yildirim
>
> The RabbitMQ connector does not offer any fault tolerance guarantees right
> now, because it does not participate in the checkpointing.
> We should integrate it in a similar was as the {{FlinkKafkaConsumer}} is
> integrated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
